ANNIHILATOR VARIETIES, ADDUCED REPRESENTATIONS, WHITTAKER FUNCTIONALS, AND RANK FOR UNITARY REPRESENTATIONS OF GL(n)

In this paper, we study irreducible unitary representations of GLn(R) and prove a number of results. Our first result establishes a precise connection between the annihilator of a representation and the existence of degenerate Whittaker functionals, thereby generalizing results of Kostant, Matumoto and others. Our second result relates the annihilator to the sequence of adduced representations, as defined in this setting by one of the authors. Based on those results, we suggest a new notion of rank of a smooth admissible representation of GLn(R), which for unitarizable representations refines Howe’s notion of rank. Our third result computes the adduced representations for (almost) all irreducible unitary representations in terms of the Vogan classification. We also indicate briefly the analogous results over complex and p-adic fields.